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ABSTRACT 


This report presents the results of an evaluation of 
NASA's Applications Pilot Test (APT) and Applications System 
Verification and Transfer (AVST) Programs managed by the 
Office of Space and Terrestrial Applications (OSTA) . These 
programs sponsor cooperative projects between NASA and poten- 
tial users of remote sensing (primarily Landsat) technology 
from federal and state government and the private sector. 
Fifteen specific projects, seven APT's and eight ASVT's, are 
examined as mechanisms for technology development, test, and 
transfer by comparing their results against stated objectives, 
interviews with project managers from NASA field centers and 
user agency representatives provide the basis for project 
evaluation from NASA and user perspectives. The intent of 
this study is not to provide project-specific findings or 
ratings; but is to find program-wide trends and arrive at con- 
clusions and recommendations for the entire programs, based 
upon examination of individual projects. Results of this 
evaluation are provided as guidance for the conduct of future 
APT and ASVT projects. 
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SECTION 1. INTRODUCTION 



SECTION 1. INTRODUCTION 


NASA's Applications Systems Verification and Transfer 
(ASVT) and Application Pilot Test (APT) Programs have been 
underway for several years. A number of specific projects 
within each program have been completed or are near comple- 
tion. Until now, a formal, structured evaluation of these 
programs has not been performed but is appropriate for pro- 
viding NASA Headquarters with guidance and direction in the 
future of these programs. In this study, OAO Corporation 
(OAOCO) has examined 15 ASVT and APT projects managed from 
five NASA field centers to evaluate the results of the joint 
application project concept. It was designed to determine the 
degree to which stated project objectives and goals were met, 
to identify problem areas, and to assess user satisfaction and 
project accomplishments. Information to support this task was 
gathered from interviews with each NASA project manager and 
selected project participants from the user community. The 
findings, conclusions, and recommendations of this study are 
based upon an evaluation of this information. 

1.1 BACKGROUND 

NASA's charter defines it as a space research and devel- 
opment agency and in the conduct of these efforts NASA has 
played an active role in the test and in the transfer of 
remote sensing technology to representative user clientele. 
Satellite-derived information, primarily from Landsat, has 
been the major focus of these technology test and transfer 
efforts to date. In the mid-seventies, NASA initiated a joint 
user development and demonstration program to promote applica- 
tions of this technology in the operational environment of the 



user conmunity. Projects in this program were cooperative 
efforts in which an application of Landsat technology was 
developed and demonstrated for operational implementation by 
the user participants. 

In the late-seventies , NASA recognized that the many 
applications of Landsat technology differed significantly in 
the level of research and development effort required. The 
feasibility of some applications had to be tested and their 
techniques developed and refined, while other applications 
involved techniques which had been essentially proven in pre- 
vious studies. As a result, NASA divided the original joint 
user development and demonstration program into two components 
beginning in fiscal year 1979: The Applications Pilot Test 
(APT) and the Applications System Verification and Transfer 
(ASVT) Programs. The APT Program, conducted by the research 
divisions, was to involve projects in which there was a signi- 
ficant amount of research and development activity to develop 
and test the technical feasibility of an application before 
technology transfer was undertaken. The new ASVT program, 
conducted by the technology transfer division, was to involve 
projects in which an essentially proven application was veri- 
fied and the majority of effort was expended in the technology 
transfer process. 

This study evaluates these two programs as technology 
transfer mechanisms by examining fifteen specific projects 
.(see section 2.2) in the two programs; seven APT's and eight 
ASVT's. The reason these two programs were evaluated in 
parallel is that both programs centered around the use of 
remote sensing (primarily Landsat) technology and both 
involved joint projects designed to meet the needs of specific 
potential users. 



1.2 STUDY OBJECTIVE 


The overall objective of this study has been to evaluate 
the APT and ASVT projects (listed in section 2.2) as a mech- 
anism for technology development, test, and transfer by 
comparing their results against the project objectives. 

The evaluation examined: 

o what the participants (user and NASA) gained from the 
project, including the extent to which the original 
project objectives were met. 

o to what extent the participants and other users 

assimilated the new techniques and the benefits 
therefrom. 

o areas of additional technology transfer that have 

been or could be pursued with the project results. 

Moreover, this study has gathered information on user 
satisfaction, identified problem areas and assessed what NASA 
has learned as a result of the joint project concept. 

While the objective of this study has been the evaluation 
of APT and ASVT projects, it was not intended to provide pro- 
ject-specific findings or a ranking or rating of individual 
projects. The intent of this study was to find program-wide 
trends and arrive at conclusions and recommendations for the 
entire programs based upon examination of the individual pro- 
jects. The variety of methodologies, procedures, and condi- 
tions under which the projects were undertaken provided a 
credible means to identify the positive and negative factors 
affecting the projects and to recognize successful and unsuc- 
cessful techniques used in conducting them. 



1.3 STUDY METHODOLOGY 


AS specified in the Task Order authorizing this evalua- 
tion, the primary source of information for this evaluation 
was in-person interviews with NASA field center project 
managers and project participants from the user community. 
The compilation and summarization of the information gathered 
from these interviews are the basis for the findings, conclu- 
sions, and recommendations presented in this report. The 
following activities were performed in preparing for, conduct- 
ing, and evaluating the results of these interviews: 

a. The NASA technical officer provided OAOCO with avail- 
able documentation for the projects to be evaluated. 
A variety of documentation from NASA, its contrac- 
tors, and user participants was provided including 
project plans, periodic progress reports, individual 
study phase plans, and status summaries. OAOCO 
reviewed this documentation to gain familarity with 
individual project's objectives, activities, and 
schedules. This review provided background informa- 
tion necessary for the interview process and high- 
lighted the similarities and differences among the 
individual projects as well as trends within the two 
programs. 

b. Two questionnaires were prepared; one for use in 
interviewing NASA project managers at the field cen- 

' ters, and the second for interviewing user agency 

participants. These questionnaires were developed 
for use by OAOCO personnel to help structure and 
standardize the interviews. They provided a format 



for obtaining pertinent information to answer ques- 
tions such as the following presented in the original 
statement of work: 


(1) What is the extent to which project objectives 
are being (were) met? 

(2) What are the users learning or have learned from 
their projects? 

(3) What is NASA learning or has learned from the 
joint project concept? 

(4) What technique developments are being derived 
from these projects? 

(5) What is the extent to which technology transfer 
was achieved, or how it could be pursued with 
project results? 

(6) What is the status and extent of each project's 
documentation? 


(7) 


What is the extent to which 
the techniques occu^ed and 
derived? 


user 

what 


acceptance of 
benefits were 


Both questionnaires were edited and approved by the 

techical officer before use in the interviews. The 
A 

NASA questionnaire appears in Appendix A and user 
questionnaire in Appendix B. 

Each NASA project manager (listed in Section' 2.2) was 
interviewed by OAOCO personnel at his(her) field cen- 
ter. These interviews were usually about 2 hours in 
duration and were tape recorded to allow for thorough 



review of the information obtained after all the 
interviews were completed. The interviews were 
primarily in discussion format in which the project 
managers were encouraged to talk about significant 
aspects of their projects in impromptu fashion, after 
which the interviewer asked specific questions until 
he had covered all the issues presented in the ques- 
tionnaire. By design, each NASA project manager was 
interviewed before questioning any corresponding 
user agency personnel. 

The task plan requested that telephone interviews be 
conducted for users in each project, with selected 
in-person interviews performed only for high 
priority projects. However, OAOCO chose to visit one 
or more users for all but two of the projects. This 
decision, with concurrence by the technical officer, 
was based upon the success of the in-person inter- 
views with the NASA project managers. The coopera- 
tion and interest shown in those in-person interviews 
demonstrated that this technique was more effective 
than telephone interviews would be. 


The user interviews were initiated by a letter of 
introduction from NASA Headquarters in most cases. 
Interviews were conducted with the project coor- 
dinator from the user community if one had been 
specifically designated. Whenever feasible, and 
especially when more than one user agency was in- 
volved, several users were invited to participate in 
the interviews. The users interviewed for each pro- 
ject are listed in Table 1 and 2. These interviews 
were usually about 1% hours in duration and were 



also tape recorded. Again, the users were given the 
opportunity to discuss those aspects of the projects 
of greatest interest to them followed by specific 
questions as outlined in the questionnaire. 


e. The information obtained from each of the interviews 
was compiled and examined. These results were evalu- 
ated as a whole to arrive at some general trends or 
overall significant findings. A draft report of 
these results and findings was presented to NASA OSTA 
personnel for review. 

f. This final report was prepared based upon further 
expansion and discussion of the results presented in 
draft report, with the incorporation of the comments 
and ideas from the NASA review. 



SECTION 2 


APT AND ASVT PROJECTS EVALUATED 



SECTION 2 : APT AND ASVT PROJECTS EVALUATED 


2.1 INTRODUCTION 

This section contains a brief sununary and description of 
the projects evaluted in this study. The list of projects to 
be evaluated was provided by the NASA technical officer and 
included those APT's and ASVT's which have been completed or are 
near completion. Several projects more recently initiated 
were not evaluated because they were at least two years from 
completion and only interim results were available. 

The descriptions which follow include the official 
project title, the user participants, and a summary of the 
project activities. In addition, the NASA field center from 
which the project was managed and the NASA project manager are 
listed as well as the fiscal years in which the project was 
conducted. The time periods shown here reflect those stated in 
project plans, although actual schedules were often signifi- 
cantly different due to delays in project initiation or exten- 
sions beyond the original schedule. Finally, the contractors 
acquired specifically to support individual projects are 
listed. General, on-site support service contractors are not 
shown. 

Funding by NASA and the user participants is not pre- 
sented in these project summaries because the technical 
bfficer did not request it as a factor in the analysis of the 
projects. Due to the variety of conditions and circumstances 
in which the projects were undertaken, it would be very diffi- 
cult to asess whether funding levels were too high or low for 
specific projects; or to weigh project expenditues against 



accomplishments. Moreover, funding commitments did change 
during the course of some projects, making it difficult to 
obtain an exact tally of project expenditures, (A general 
discussion of the topic of funding is discussed in section 4.2. ) 

2. 2 PROJECTS EVALUTED 

Summary descriptions of the seven APT's and eight ASVT's 
evaluated in this study are listed below. 



PROJECT NAME Automated Cotton Acreage Inventory System 

Based On Landsat Data APT 

USER 

PARTICIPANT (S) Cotton, Incorporated 


PROJECT 

DESCRIPTION A cotton acreage inventory technique using 

computer analysis of Landsat digital data 
will be developed and tested in six domestic 
and one foreign test sites in major cotton 
producing areas. The results of these 
analyses will be evaluated with respect to 
accuracy, timeliness, and cost-effec- 
tiveness. If the technique meets the 
requirements of Cotton, Inc. , it will be 
implemented in an operational program to 
inventory cotton acreage. 

NASA FIELD 

CENTER/PROJECT 

MANAGER National Space Technology Laboratories 

(NSTL)-Earth Resources Laboratory/R. Griffin 


FISCAL YEARS '79- '81 


CONTRACTOR(S) 


NONE 



PROJECT 

NAME 


USER 

PARTICIPANT (S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YERS 


CONTRACTOR ( S ) 


Census Urbanised Area Delineation APT 


Bureau of Census 


The use of computer analysis of Landsat MSS 
data will be tested for census applica- 
tions. Techniques will be developed for 
defining selected urban land cover 
categories for the perimeter of five U.S. 
urbanized areas. The accuracy of 

information derived from these techniques 
will be evaluated and procedures for 
integrating this information into existing 
census data will be developed. Useful 
technology will be implemented for opera- 
tional use by the Bureau of Census for 
periodic delineation and updating of 
urbanized areas. 


Goddard Space Flight Center/D. Toll 

•77 - '80 

GENERAL ELECTRIC 

JET PROPULSION LABORATORY 



PROJECT NAME 


USER 

PARTICIPANT (S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEAR(S) 


CONTRACTOR(S) 


Forest Resource Information System APT 

St. Regis Paper Company (Southern Timberlands 
Division) 


The feasibility of utilizing Landsat MSS 
data as a viable contributor to an opera- 
tional forest resource information system 
will be tested, Landsat MSS digital data 
will be analyzed to determine forest cover 
information for test sites in the holdings 
of St. Regis Paper Company. If the utility 
of Landsat-der ived information is success- 
fully demonstrated, the capability to 
perform Landsat analysis will be estab- 
lished within St. Regis. This capability 
will be integrated into and become a signi- 
ficant data source in a computerized forest 
resource information management system. 


Johnson Space Center/R. Joosten 

•77 - '80 

Purdue University 



PROJECT NAME 


USER 

PARTICIPANT (SV 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEARS 


CONTRACTOR(S) 




Irrigated Lands Assessment for Water 
Management APT 

California Dept, of Water Resources (DWR) 

Digital analysis and image interpretation 
techniques of Landsat products will be 
examined for use in delineating irrigated 
and non-irr igated cropland as well as dis- 
cerning irrigated crop types. Analysis of 
Landsat data will be performed on test sites 
throughout California using various levels 
of sophistication. Each of the techniques 
developed and tested will be evaluated by 
California DWR for suitability of incorpo- 
ration into their operational water manage- 
ment models and decisions. Training and 
assistance in the implementation of viable 
techniques will be provided to California 
DWR. 


Ames Research Center/E. Bauer 
'78 - *82 

Universities of California at Berkley and 
Santa Barbara 



PROJECT NAME 


USER 

PARTICIPANT (S) 


PROJECT 

description 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEAR(S) 


CONTRACTOR (S) 


Land Cover Change Detection and Update APT 


U.S. Geological Survey (Geography Program) 


Techniques for detecting changes in land 
cover using Landsat MSS digital data will be 
developed and demonstrated on five test 
sites in Louisiana and Wyoming. The value 
of these techniques for updating (locating 
change) in the land use maps produced in the 
USGS Land Use and Data Analysis (LUDA) Pro- 
gram will be tested. Based upon the verifi- 
cation of the utility of these techniques, 
the capability to perform these analysis 
will be established within the Geography 
Program of the USGS. 


National Space Technology Laboratories 
(NSTL) - Earth Resources Laboratory/A. 
Joyce 

’77 - '79 

NONE 



PROJECT NAME 


USER 

PARTICIPANT (S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEARS 


eONTRACTOR(S) 


Landsat Based Automated Resource Inventory 
for the Navajo Reservation (APT) 


Navajo Indian Nation 


An automated resource base inventory and 
update system based on Landsat digital data 
and oriented to Navajo requirements will be 
developed and tested. Demonstrations of 
computer implemented analysis of Landsat 
MSS data will be conducted for various 
applications using test sites on the Navajo 
reservation (New Mexico, Arizona, and 
Utah). Based upon an evaluation of the 
system's capabilities by the Navajo Nation, 
it will be implemented on Navajo facilities 
for use in operational resource management 
activities. 


National Space Technology Laboratories 
(NSTL) - Earth Resources Laboratory/P. 
Conner 

•78 - '80 

Navajo Community College 



PROJECT NAME Wildland Vegetation Resource Inventory APT 


USER 

PARTICIPANT (S ) Bureau of Land Management (BLM) ( Denver Serv ice 

Center) 


PROJECT 

DESCRIPTION An interactive, automated wildland vegeta- 

tion inventory system based upon remotely 
sensed data and oriented to BLM management 
requirements will be developed and tested. 
Landsat MSS and other types of remotely 
sensed data will be used for wildland 
vegetation inventories in test sites in 
Alaska, Arizona, and Idaho. This work will 
include demonstration, evaluation, and also 
operational testing of the techniques which 
BLM has chosen to adopt for use in its 
resource management activities. 

NASA FIELD 

CENTER/PROJECT 

MANAGER Johnson Space Center/K. Hancock 


FISCAL YEAR(S) '77 - '79 


CONTRACTOR { S ) Electromagnetic Systems Laboratories, Inc. 

(ESL) 



PROJECT NAME 


USER 

PARTICIPANT(S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEAR(S) 


CONTRACTOR(S) 


Appalachian Lineaments ASVT 


Appalachian Regional Commission and 
agencies and universities in seven member 
states: New York, Pennsylvania, Ohio, 
Virginia, West Virginia, Kentucky, and 
Tennessee. 


Enhanced Landsat imagery products will be 
generated for three test sites within the 
seven state region for use in delineation of 
fracture zones (lineaments). The most 
useful seasons and enhancement procedures 
will be determined and used by participants 
to identify fracture zones. An optimal set 
of procedures to utilize Landsat-der ived 
information will be developed and docu- 
mented. The value of information derived 
from Landsat products in support of ongoing 
state geologic programs will be determined 
and the use of this information will be 
supported. 


Goddard Space Flight Center/H. Blodgett 
'78 - ’80 
General Electric 



PROJECT NAME 


ICEWARN ASVT 


USER 

PARTICIPANT (S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEAR(S) 


CONTRACTOR(S) 


U.S. Coast Guard, National Oceanic and 
Atmospheric Administration 


An all-weather, near real-time, airborne 
ice information system to aid in the exten- 
sion of winter navigation on the Great Lakes 
will be developed and demonstrated. This 
side-looking airborne radar (SLAR) system 
will collect image products which can be 
interpreted to show ice thickness and sea 
conditions. Images on which this informa- 
tion has been delineated can be transmitted 
to ships at sea for use in navigation. If 
proven successful, this system will be 
implemented and operated by the Coast 
Guard. 


Lewis Research Center/H. Mark 

'74 - *79 

NONE 



PROJECT NAME Operational Applications of Satellite Snow- 

cover Observations ASVT 

USER 

PARTICIPANT ( S ) Federal Agencies; Soil Conservation 

Service, U.S. Geological Survey, U.S. 
Bureau of Reclamation, Bonneville Power 
Administration, U.S. Army Corps of 

Engineers, and National Oceanic and Atmo- 
spheric Administration. 

State Agencies; California Dept, of Water 
Resources, Colorado Div. of Water 

Resources, and Arizona Salt River Project. 

A technique will be developed and docu- 
mented to map snowcover on image products 

from Landsat and NOAA satellites. 
Satellite imagery of test areas in Arizona, 
California, Colorado, and the Pacific 

Northwest will be provided to user partici- 
pants in a timely fashion (about 72 hour 
delivery) so that snowcover information can 
be incorporated into traditional runoff 
prediction models. The benefit of snow- 
cover information in various runoff models 
will be assessed and, when feasible, the use 
of snowcover information will be imple- 
mented in operational programs. 

NASA FIELD 
CENTER/PROJECT 

MANAGER Goddard Space Flight Center/A. Rango 


PROJECT 

DESCRIPTION 



FISCAL YEAR(S) 
CONTRACTOR ( S ) 


'75 - '78 

NOAA, Ecosystems International, Inc. 
Sierra Hydrotech, Inc. 



PROJECT NAME 


USER 

PARTICIPANT(S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEARS 


CONTRACTOR ( S ) 


Pacific Northwest Project - Landsat Appli- 
cations Program ASVT 

Pacific Northwest Regional Commission, 
state agencies and universities in 
Washington, Oregon, and Idaho. (Technology 
transfer support to NASA from USGS-EROS and 
Geography programs. ) 


Based upon the favorable results of a number 
of cooperative demonstration projects 
covering a variety of applications and 
involving over 30 user participants in the 
three state area, an operational capability 
for analysis and utlization of Landsat data 
will be established in each state. NASA 
will assist each state in establishing, 
testing, and implementing a capability, 
using available state resources, which is 
designed for use by state agencies in 
managing natural resources. 


Ames Research Center/D. Wilson 
•79 - '81 

Electromagnetic Systems Laboratory, Inc. 
(ESL) 



PROJECT NAME 

Surface Water Detection and Mapping (DAM) 
ASVT 

USER 

PARTICIPANT (S) 

U.S. Army Corps of Engineers (regional and 
district offices), selected state agencies 
in the 50 states. 

PROJECT 

DESCRIPTION 

The Detection and Mapping (DAM) Package 
developed by NASA for identifying surface 
water using Landsat MSS digital data will be 
upgraded and maintained for operational 
use. This procedure will be implemented in 
the national water impoundment inventory 
which supports the National Program of 
Inspection of Dams Legislation. Products 


showing surface water on a quad map basis 
will be generated for all 50 states and 
field personnel will evaluate and report on 
the accuracy and utility of the information 
provided. 

NASA FIELD 
CENTER/PROJECT 

MANAGER 

Johnson Space Center/D. Amesbury 

FISCAL YEARS 

00 

m 

00 

o 

CONTRACTOR { S ) 

NONE 



PROJECT NAME 


USER 

PARTICIPANT (S) 


PROJECT 

DESCRIPTION 


Texas Natural Resources Inventory and 
Monitoring System ASVT 

Thirteen Texas State Agencies: Department 

of Water Resources, General Land Office, 
Air Control Board, Bureau of Economic 
Geology, Railroad Commission, Department of 
Agriculture, Forest Service, Industrial 
Commission, Department of Health, Depart- 
ment of Highways and Public Transportation, 
Parks and Wildlife Department, State Soil 
and Water Conservation Board, and the 
Coastal and Marine Council. (These 13 
operate the Texas Natural Resources Infor- 
mation System (TNRIS) Task Force). 


This project will develop an operational 
remote sensing capability, utilizing a mix 
of manual and computer-assisted techniques, 
to inventory and monitor natural resources 
in the state of Texas. This capability will 
be tested in a variety of demonstration 
applications involving several state 
agencies. Those elements of this capa- 
bility which the agencies judge to be useful 
and cost-effective will be implemented in 
the Texas Natural Resources Inventory and 
Monitoring System, a computerized geo- 
graphic information system. 



NASA FIELD 
CENTER/PROJECT 

MANAGER Johnson Space Center/L. Child 


FISCAL YEARS '78 - '80 

CONTRACTOR (S) Texas A&M University 



PROJECT NAME 


USER 

PARTICIPANT(S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEARS 


Vegetation Resource Base Inventory and 
Monitoring ASVT 

National Park Service (NPS) (Denver Service 
Center) 


Computer -implemented techniques to derive 
geograph ically-re fere need vegetat ion/land 
cover information from Landsat MSS data 
will be developed and demonstrated in three 
National Parks having diverse vegetation, 
climate, and topography. Based upon NPS 
approval, a capability (hardware and soft- 
ware) to use such techniques will be estab- 
lished within the NPS to support opera- 
tional park resource management activities. 


National Space Technology Laboratories 
(NSTL)-Earth Resources Laborat ion/w. Cibula 

'79 - '81 


CONTRACTOR ( S ) 


NONE 



PROJECT NAME 


USER 

PARTICIPANT (S) 


PROJECT 

DESCRIPTION 


NASA FIELD 
CENTER/PROJECT 
MANAGER 


FISCAL YEARS 


CONTRACTOR(S) 


Water Management and Control ASVT 


U.S. Army Corps of Engineers (Hydrologic 
Engineering Center/Davis, California) 


A relatively automated, simple-to-operate 
technique to utilize Landsat data for 
acquiring the land use or surface cover 
information required for hydrologic 
engineering models employed in flood 
control and waterworks planning, design, 
and management will be developed and 
demonstrated. Six watersheds will be 
studied to provide technique development and 
training in these procedures. The final 
technique will be evaluated, documented, 
and made available for use by Corps district 
offices. 


Goddard Space Flight Center/A. Rango 
'77 - '79 

University of California at Davis, Bendix, 
Battelle, University of Maryland 



2.3 


NASA AND USER PERSONNEL INTERVIEWED. 


Tables 1 and 2 which follow list the personnel from the 
NASA field centers and the user agency (s) who were interviewed 
in this study. For every project, the current NASA project 
manager (managers changed during the course of some projects) 
was interviewed. The persons interviewed from the user commu- 
nity were direct project participants and recommended by the 
NASA project managers. 



TABLE 2.1 PERSONS INTERVIEWED FOR THIS STUDY (APT's) 


PROJECT 

TITLE 


NASA FIELD CENTER 
PERSONNEL 


USER AGENCY(S)- 
PERSONNEL 


O AUTOMATED COTTON ACREAGE NSTL/ERL - R. GRIFFIN, 

INVENTORY SYSTEM BASED ON A. JOYCE 

LANDSAT DATA 


COTTON, INC. 

J. TULLOS, L. SHAW 


o CENSUS/URBANIZED 
AREA DELINEATION 

O FOREST RESOURCES 
INFORMATION SYSTEM 

O IRRIGATED 

LANDS ASSESSMENT FOR 
WATER MANAGEMENT 

O LAND COVER CHANGE 

t 

DETECTION & UPDATE 


GSFC - D. TOLL 


JSC - R. JOOSTEN 


ARC - E. BAUER 


NSTL/ERL - A. JOYCE 


BUREAU OF CENSUS - 
J. SILVER* 

ST. REGIS PAPER CO.- 
B. BARKER 

CA. DEPT. OF WATER 
RESOURCES-G. SAW- 
YER, C. FERCHAUD 

U.S. GEOLOGICAL SURVEY 
D. WITMER, V. MILAZ7.0, 
S. GUPTILL 


* NOT PERSONALLY INTERVIEWED 



TABLE 2.1 PERSONS INTERVIEWED FOR THIS STUDY (APT's) (cont. ) 


LANDSAT BASED AUTOMATED RESOURCE 

NSTL/ERL - P. CONNOR 

NAVAJO TRIBE- 

INVENTORY FOR THE 


E. BENALLY 

NAVAJO RESERVATION 



WILDLAND VEGETATION 

JSC - K. HANCOCK 

BUREAU OF LAND MANAGEMENT 

RESOURCE INVENTORY 


B. BONNER 



TABLE 2.2 PERSONS INTERVIEWED FOR THIS STUDY (ASVT's) 


PROJECT 

NASA FIELD CENTER 

USER AGENCY (S)- 

TITLE 

PERSONNEL 

PERSONNEL 

APPALACHIAN LINEAMENTS 

GSFC - H. BLODGETT 

APPALACHIAN REGIONAL 
COMMISION-J. DEMCHALK 

ICEWARN 

LeRC - H. MARK 

U.S. COAST GUARD-R. KNAPP* 

OPERATIONAL APPLICATIONS 
OF SATELLITE SNOWCOVER 
OBSERVATIONS 

GSFC - A. RANGO 

CA. DEPT. OF WATER RESOURCES 
J. PARDEE, J. BROWN, N. 
PETERSON; 

SIERRA HYDROTECH- , 

J. HANNAFORD 

PACIFIC NORTHWEST 
PROJECT- LAND SAT 
APPLICATIONS PROGRAM 

ARC - D. WILSON 

IDAHO DEPT. OF WATER 
RESOURCES-K. JOHNSON; 
PACIFIC NORTHWEST REG. 
COMM. TASK FORCE- A. PORTER 

SURFACE WATER DETECTION 

JSC - D. AMESBURY 

U.S. ARMY CORPS OF ENGINEERS- 

AND MAPPING (DAM) 


I. McKIM 

TEXAS NATURAL RESOURCES 
INVENTORY AND MONITORING 
SYSTEM 

JSC - L. CHILDS 

TEXAS DEPT. OF WATER 
RESOURCES-S. McCULLOCH 


♦NOT PERSONALLY INTERVIEWED 



TABLE 2.2 PERSONS INTERVIEWED FOR THIS STUDY (ASVT's) (cont. ) 


PROJECT 

TITLE 

O VEGETATION RESOURCE 

BASE INVENTORY AND MONITORING 


O WATER MANAGEMENT AND CONTROL 


NASA FIELD CENTER 
PERSONNEL 


USER AGENCY (S) - 
PERSONNEL 


NSTL - W. CIBULA, 
P. CONNER 


NATIONAL PARK SERVICE- 
M. NYQUIST 


GSFC - A. RANGO 


U.S. ARMY CORPS OF ENGINEERS- 
A. FELDMAN, B. CERMAK 



SECTION 3. SPECIFIC FINDINGS IN RESPONSE TO 
TASK ORDER QUESTIONS 



SECTION 3. SPECIFIC FINDINGS IN RESPONSE TO 
TASK ORDER QUESTIONS 


3.1 INTRODUCTION 

The interviews conducted for this evaluation were used as a 
mechanism to gather specific information about the fifteen APT 
and ASVT projects. The questionnar ies for interviewing were 
developed to help gather the information in a standard format. 
This information can now be used to answer several specific 
questions posed in the original Task Order authorizing this 
study. This section provides answers to these specific 
questions . 

Additional observations and findings based upon information 
gathered in the interviews is provided in Section 4. 

3.2 FINDINGS IN RESPONSE TO SPECIFIC QUESTIONS 

Seven specific questions pertinent to an evaluation of the APT 
and ASVT projects were posed in the task plan. These ques- 
tions are listed below, followed by a response to each based 
upon the information gathered in the interview. 

(1) What is the extent to which project objectives are 
being (were) met? 

(2) What are the users learning or have learned from 
their projects? 

(3) What is NASA learning or has learned from the joint 
project concept? 



(4) What technique developments are being derived from 
these projects? 

(5) What is the extent to which technology transfer was 
achieved, or how it could be pursued with project 
results? 

(6) What is the status and extent of each project's 
documentation? 

(7) What is the extent to which user acceptance of the 
techniques occurred and what benefits were derived? 



RESPONSES TO QUESTIONS 


1. What is the extent to which project objectives are being 
(were) met? 

There are significant differences in the "objectives of the 
various ASVT and APT projects evaluated. Objectives of most 
projects focus on plans to: 

a. develop, test the feasibility of, or demonstrate the 
use of Landsat data applied to some discipline, 

or, 

b. design and implement an automated resource manage- 
ment information system with Landsat as a significant 
data source. 

Some projects were designed to establish a Landsat capability 
with a user first, followed by a user evaluation of the 
utility of that system. By contrast, other projects were 
designed to have NASA demonstrate and the user evaluate the 
technology; subsequently the user would make a decision 
whether or not to establish his own Landsat capability. 

Approximately a third of the projects have met their objec- 
tives as stated in the project plans. In addition, another 
third of the projects are still underway and even though prob- 
lems exist, have the potential of meeting all of their stated 
objectives. The objectives in the remainder of the projects 
were only partially attained or not accomplished. Reasons 
that project objectives have not been met include the in- 
ability of the technology to meet the needs of the user, time 
constraints on project activities, and the inability of users 
to make or maintain commitments to their projects. 



2. What are the users learning or have learned from their 
projects? 

Every user interviewed believes that he was given an excellent 
opportunity to fairly evaluate the utility of remote sensing, 
particularly Landsat, in his resource management activities. 
Users now feel they have an appreciation for the capabilities 
of the technology as well as its limitations. The consensus 
among the user community was that they learned, through first- 
hand experience, that Landsat alone is not the answer to their 
resource management problems. However, when it is used in 
conjunction with other data sources, it can be of great value 
because it provides information which, until now, was not 
readily available through conventional means. Of particular 
value to many users was the large area, synoptic coverage of 
Landsat, as well as its repetitiveness. Some users found that 
Landsat data provided inventory information comparable to that 
from conventional methods but at a c’heaper cost, while other 
users derived more accurate or complete inventories because of 
its coverage of large or inaccessible areas. 

3. What is NASA learning or has learned from the joint pro- 
ject concept? 

a. Users generally agree that information derived from 
Landsat technology is most useful and appropriate when 
incorporated into a larger information system utilizing 
information from various sources. Seldom is Landsat- 
derived information of enough value by itself to warrant 
the user community investing in the technology. 

b. Landsat generally provides these users with a new 
source of information which allows them to better manage 
resources. It does not replace traditional means of 



information gathering, therefore the incorporation of 
Landsat technology, while it does improve the level of 
service, will usually not decrease overall costs of 
resource management. Landsat often provides users with 
an affordable method of performing large area inventories 
which they would otherwise have difficulty accom- 
plishing. In addition, it provides data in a standard 
format with relatively frequent repetitive coverage. 

c. The current ground system with its data delivery prob- 
lems and the uncertainty of the future satellite config- 
urations and launch dates are serious impediments to the 
adoption of Landsat technology. Approval by user manage- 
ment personnel for Landsat system purchases are difficult 
to justify in light of these problems. In some cases, 
users have been able to support an investment in a 
Landsat capability only because it is part of a larger 
geographic information system. If a hiatus in data 
delivery occurs these programs can survive using other 
data sources. In other case(s), NASA has successfully 
transferred a technology which users would implement in 
an operational program if the data were available in a 
timely fashion. Under the current conditions of data 
delivery, however, these users cannot apply the tech- 
nology in their operational programs. 

d. The demonstration/Landsat processing phase of ASVT/ 
APT projects has almost always taken longer than origi- 
nally scheduled, although additional funding was not 
necessarily required. In some cases, NASA has been very 
ambitious in the amount of demonstration work it agreed 
to undertake, and this has resulted in the deletion of 
evaluation or verification work in the later stages of 
some projects. Such deletions were felt to be detri- 
mental to the validity of the test. 



e. Project activities which were recognized as research 
and development at the start were very difficult to 
schedule stringently. R&D is nearly always an iterative 
process which can and, in these projects, has taken sig- 
nificantly longer than planned. 

f. For a variety of reasons, most of the projects have 
taken longer than originally planned. Most of the users 
in federal and state governments were not bothered by 
this and some admit their agency caused a good share of 
the delays. However, participants in the two projects 
involving private industry users were very conscious of 
schedules and deadlines. Private industry was less 
tolerant of delays and this must be recognized in project 
planning. 

g. The phasing of project activities to correspond with 
the users' activities and requirements was found to be 
important. For example, an evaluation of the technology 
by a user should be completed such that the user has the 
information to make a GO/NO-GO decision in concert with 
his budget cycles. Similarly, the timing of contracting 
procedures between NASA and its contractors was found to 
be a significant consideration in project planning. 

4. What technique developments are being derived from these 
projects? 

Every project has resulted in some technique development due 
to the specific information needs of its applications. Even 
though many of the ASVT’s employed what may be called 
"proven" technology, such as landcover classification, the 
idiosyncrasies of each user's applications required at least 
minor modifications to "standard" procedures. In addition to 



this specialization of standard techniques to particular 
applications, several significant advances were made during 
apt's in the field of Landsat data analysis: 

a. Several projects have made contributions in the use 
of Landsat-der ived information in a geographic informa- 
tion system, which stores and manipulates information 
from several sources. Developments in this field include 
improved techniques for registration and geometric cor- 
rection of Landsat data, software to merge Landsat data 
in "grid" format with other data types in "polygon" for- 
mat, and new applications which integrate Landsat and 
other data types to solve specific resource management 
problems. 

b. The use of Landsat in detecting changes in landcover 
has been improved through the development of several new 
procedures, including "automated" techniques for regis- 
tering different Landsat scenes and recognizing changes 
in raw data. The limitations of change detection tech- 
niques were also extensively explored. 

c. Sampling methodologies were developed and refined 
for using information from Landsat in conjunction with 
other sources like aircraft photography and ground sur- 
veys. 

5. What is the extent to which technology transfer was 
achieved, or how it could be pursued with project results? 

# 

Of the projects evaluated, over half have resulted in user(s) 
establishing some type of in-house capability to analyze Land- 
sat data. However, only two of these users currently, or in 
the very near future, will utilize Landsat-der ived information 



in an operational setting. The remainder are using Landsat- 
derived information in a research setting which could result 
in operational implementation at a later date. There are many 
complex and inter-related issues which are keeping the remain- 
der of these Landsat processing capabilities from operational 
usage. Some of these issues include user agency politics^ 
lack of faith in the technology by user personnel in the 
operational programs, lack of timely Landsat data, inability 
of the technology to meet the needs of operational programs, 
lack of funding in the user agency to implement an operational 
program and lack of user involvement throughout a project. 

Of the projects in which a capability to analyze data has not 
been established to date, there are three projects in which it 
is too early to predict the outcome, although significant 
impediments exist. The remainder of these have been completed 
or terminated and there was very little, if any, success in 
establishing a user capability to utilize Landsat data to 
satisfy the originally stated objectives. Reasons that these 
projects did not result in the establishment of a user cap- 
ability to analyze Landsat data include the inability of users 
to maintain commitments to their projects; lack of a mandate 
for the users (hence no funding) to conduct an on-going, 
operational program which uses the technology; and the 
inability of the technology to meet the information require- 
ments of the users. 

6. What is the status and extent of each project's documenta- 
tion? 

It was found that there is very little final project documen- 
tation for ASVTs and APTs. Some of the projects have recently 
been completed or are still underway, due, in several cases. 



to schedule slippages. Still, project documentation has often 
lagged behind the completion of technical activities. The 
most complete and up-to-date documentation generally exists 
for those projects in which NASA has contracted with universi- 
ties or off-site contractors to conduct the activities. Pro- 
ject plans usually called for progress and final reports to be 
prepared jointly by NASA and the user but the specifics of 
these arrangements were not clearly stated. Because many of 
the projects took longer than originally scheduled, funds 
originally designated for documentation were depleted before 
the documentation activities started. 

In addition to progress reports and final reports about a 
third of the projects have produced documentation of a tech- 
nique or procedure developed as part of the project. These 
documents can be used in the transfer process to other poten- 
tial users of the technology. Most of the software developed 
or utilized in these projects will be or already has been en- 
tered into COSMIC (Computer Software Management and Informa- 
tion Center -University of Georgia). Some NASA project man- 
agers and user representatives have published or are planning 
to publish results of their projects in scientific journals. 

7. What is the extent to which user acceptance of the tech- 
niques occurred and what benefits were derived? 

A vast majority of the users with immediate involvement in 
these projects stated that the technology developed or demon- 
strated is of definite value to their resource management 
problems. The joint demonstration phase was very thorough and 
convincing to the user participants in most projects. In only 
one project did the user participants state that the tech- 
nology could not provide accurate enough information to meet 



the needs of their application. However, transferring or 
disseminating the technology from the experimental, test pro- 
ject environment to operational usage in the user agencies was 
much harder to accomplish. Most of the user project partici- 
pants were convinced of the utility of the technology but 
their counterparts in operational programs had not accepted 
it, sometimes because they were not given an adequate oppor- 
tunity to evaluate it. 

In any event, every user emphasized that some benefit was 
derived from involvement in the program. As a minimum, the 
users gained an education and awareness in the use of the 
technology for solving their resource management problems. 
This awareness might spark interest in pursuing involvement in 
improved, future satellite systems whose data may better 
satisfy user information needs. In addition, at least half of 
the user agencies are now equipped with a capabiity to analyze 
remotely sensed data, primarily Landsat, in experimental or 
operational programs. Of these, two users have gone a step 
further and will be deriving the benefits from utilizing their 
capabilities in operational activities in their agencies. 
Three users could have the same capability once their projects 
have been completed. 
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SECTION 4. ADDITIONAL OBSERVATIONS AND FINDINGS 


4.1 INTRODUCTION 

This section presents a variety of additional findings 
stenuning from examination of the interview information from 
all fifteen projects. Included are observations of both posi- 
tive and negative aspects of the program as a whole. In addi- 
tion, factors which have had significant impact (either bene- 
ficial or detrimental) on several individual projects will be 
examined. These trends have surfaced as a result of analysis 
of the data base of information from the fifteen individual 
projects. 

4.2 ADDITIONAL FINDINGS 


These observations and discussions of their significance 
appear below. In the following subsection, 4.3, many of these 
findings and some of those in section 3.2 are summarized as 
factors affecting projects in the APT and ASVT programs. 

VARIETY OF APPLICATIONS, CROSS-SECTION OF USERS 

NASA has covered a wide variety of applications in the 
ASVT and APT projects. The use of Landsat and other types of 
remotely sensed data have been demonstrated, developed, and 
tested in specific applications of forestry, agriculture, 
hydrology, geology, and urban development. The APT and ASVT 
programs have reached a broad cross-section of potential users 
in federal, state, and substate government and in the private 
and academic sectors. Project participants from the user 
agencies were very cooperative and willing to disseminate the 



results of their projects throughout their own organizations 
using briefings, demonstrations, and internal publications. 
Moreover, other potential users and the general public have 
been exposed to the applications of Landsat data in these 
projects through scientific journal articles, symposia, work- 
shops, and newspaper and television coverage. Through this 
variety of means these programs have greatly contributed to 
the general awareness of Landsat technology in many disci- 
plines. 

UNIQUE OPPORTUNITY FOR USERS 

Without exception, the users interviewed expressed 
appreciation for the opportunity to participate in their pro- 
jects. NASA provided the users with thorough, hands-on 
experience in the field of Landsat remote sensing which they 
could not have acquired any other way. Limited R&D funds 
preclude many potential users from their own development and 
testing of space-derived information systems for improved 
resource management. Regardless of the results of the tech- 
nology transfer in the projects, users have gained an aware- 
ness and education in Landsat remote sensing which will be 
valuable to their agencies in assessing future opportunities 
to use the technology. 

JOINT PROJECT CONCEPT 

The joint project concept has proven to be a viable 
method of conducting these projects. Almost universally, 
users have been satisfied with NASA field center support and 
participation in the projects. Users felt that at least orig- 
inal funding levels have been adequate and that NASA field 
centers have met their commitments to the projects. Most 
users thought that NASA field centers have been very concerned 



about their needs and have allowed adequate opportunity for 
users to make inputs into the way projects have been conduc- 
ted. In only one case was a user concerned that NASA had made 
unilateral decisions in project activities. Two other excep- 
tions occurred when users expressed some dissatisfaction with 
the time required to get their projects started due to NASA 
signature and approval cycles. 

Similarly, NASA project managers believe that the users 
met or exceeded their obligations to these joint projects in 
all but two cases. Users have been willing to apply the 
necessary resources to complete project activities, even 
though several users realized that they had originally under- 
estimated the amount of work required. Most project plans 
were specific about the responsibilities of both parties 
involved. It appeared that NASA and the user both understood 
their respective roles in the projects from the beginning. 

Another important aspect of the joint project concept 
which has surfaced is the interaction of NASA and user per- 
sonnel involved. The personal relationships which have devel- 
oped between NASA and the user participants have often played 
a significant role in project accomplishments. The mutual 
respect and friendship which developed between corresponding 
project managers was a topic often mentioned as a contributor 
to project success. Many participants thought such feelings 
provided an incentive to accomplish project goals and assure 
project success. 

POLITICAL CONSIDERATIONS 

To be successful, APT and ASVT projects must have a sound 
technical basis. It was found that political considerations 



have played a significant role in the selection and implemen- 
tation of a few of the projects. In these cases, the interest 
by NASA Headquarters or a field center in involving a particu- 
lar user may have overshadowed the technical merits of the 
project. While this does not preclude a project from being 
successful, those initiated under these circumstances have 
encountered more problems, taken longer than scheduled, or 
have required significant changes to their original plans. 

These problems may have been due to the fact that too 
little consideration was given to factors which have a 
stronger influence on project success like a sound application 
technique, genuine user interest, or the ability of users to 
make commitments to the program. 

LANDSAT DATA DELIVERY 

The Landsat data delivery system continues to have a 
detrimental effect on the outcome of ASVT and APT projects. 
It is difficult to quantify the degree to which these problems 
are hindering the implementation of Landsat technolgy, but 
most users expressed frustration with their experience in 
dealing with the data delivery system. In the 1975 to 1978 
period when most of these projects were begun, few people 
would have predicted that users would still be facing problems 
obtaining Landsat data in 1981. It is likely that most users 
felt, at the outset of their projects, that many of these 
problems would have been solved by the time they were to con- 
sider implementing the technology. In any event, data 
delivery will continue to prevent some users from utilizing 
the technology at all because of their application require- 
ments. For example, at least two of the applications devel- 
oped and demonstrated in APT and ASVT projects require Landsat 
data delivered within a week of when it was collected if it is 



to be used in operational user programs. The current data 
delivery system is unable to meet this requirement. In addi- 
tion, all users feel uneasy about what they perceive to be 
NASA's inability or perceived lack of interest in rectifying 
the data delivery situation. Their reluctance to invest in 
the technology under these conditions is understandable. 

FUTURE LANDSAT/LAND REMOTE SENSING PROGRAM 

A similar and related problem to Landsat data delivery 
which influences users adopting this technology is the uncer- 
tainty about the future of the Landsat/land remote sensing 
program as a whole. In the mid-seventies, users were assured 
of the Landsat-D generation of satellites being in operation 
or very near launch by 1980-1981. The improved capabilities 
of the new systems were designed to expand the applications of 
satellite remote sensing. However, the future of the program 
became more and more uncertain until today there is doubt in 
the minds of users as to when, or if, there will be a Landsat-D 
and how it will be configured. The transfer of responsibility 
for an operational land remote sensing system from NASA to 
NOAA has further confused the short-term (next five years) 
outlook of the program. User uncertainty about continuity 
(and even continuation) of the program puts extreme negative 
pressure on the technology transfer process. 

OPERATIONAL IMPLEMENTATION 

Among the projects evaluated, there have been several in 
which the failure to establish an operational capability to 
utilize Landsat data was due to factors beyond the present 
control of NASA. It is not uncommon in these cases that the 
technology has been successfully demonstrated and has proven 
useful to the project participants. However, because of user 



personnel changes, higher priority conmi tments, political 
considerations, or lack of funding in the user agency, the 
technology is not to be implemented in operational programs. 
While many such obstacles cannot be foreseen, NASA should 
examine users with respect to their ability to adopt the tech- 
nology. For example, NASA should not expect a user agency to 
incorporate the technology into operational programs after it 
has transferred the technology to a research branch of that 
organization which has no means to implement it operationally. 

In addition, there should be an understanding of each 
potential user's charter or mandate for resource management. 
On-going, operational user programs which can use the tech- 
nology should be identified and the factors surrounding imple- 
mentation of the technology in them addressed. When this is 
done, genuine utilization of the technology is more likely. 
For example, the user agency involved in one of the most 
successful projects was recently given a legislative mandate 
to perform an on-going inventory program for its land hold- 
ings. The user felt the existence of legislation played a 
significant role in the project success. In at least two 
other projects, viable applications were developed and demon- 
strated but the user participants had no responsibility («x 
therefore funding) to perform on-going activities utilizing 
those applications. 

IMBALANCE OF COMMITMENTS 

In some projects there was an imbalance in the commit- 
ments (money, manpower, equipment, facilities) by NASA and the 
user. Occasionally strenuous demands were placed on NASA in 
project activities and items deliverable to the user. At the 
same time user participation was significantly less. While 



this division of conunitments is not of ultimate importance, it 
is likely that the adoption and utilization of the technology 
is related to user commitment. The NASA goal in these pro- 
jects was to transfer Landsat technology into operational 
utilization by outside agencies or groups, which requires that 
such groups make a genuine commitment to that transfer process 
in the conduct of the total project. If NASA essentially pro- 
vides a Landsat analysis capability to a user who has had only 
token participation in its development, the danger exists that 
such a capability will never make the transition into the 
operational realm. On the other hand, a vested interest by 
the user, which implies a genuine belief in the technology, 
strengthens the changes for survivial of that technology. 

An example of such imbalance of commitments occurred in 
some projects in which NASA agreed to provide a Landsat remote 
sensing capability and its interface to a Geographic Informa- 
tion System (GIS) while the user devoted his efforts to devel- 
oping other aspects of the GIS - an activity he was probably 
willing to undertake regardless of its application to this 
project. The user seemed reluctant to become involved in the 
remote sensing aspects of the project, except as primarily an 
observer, possibly because of his doubts about the technology. 
What resulted from these projects were GISs with Landsat 
analysis capability and interface added to them. Each GIS, 
which was developed by the user, is a very viable resource 
management tool with or without utilizing the Landsat capa- 
bility. Hopefully, the user is now inclined to support the 
Landsat capability as part of a total system on his own. Ear- 
ly user involvement in the development of the whole system 
might have increased that inclination. 



In summary, an exact tally of commitments by each parti- 
cipant is not important and each project has its unique 
circumstances which dictate the division of responsibilities. 
However, significant user commitment is one indication of 
genuine user interest in a continued program. But, this does 
not mean projects without a tangible commitment will neces- 
sarily fail. For example, one of the most successful studies 
had very limited user commitment (in dollars or manpower) in 
the development of the Landsat analysis capability. However, 
this project was a success because the user was very convinced 
of the utility of Landsat derived information for his applica- 
tion. Yet in another project, a Landsat analysis capability 
with negligible user commitment either tangible or intangible, 
appears destined to go unused. A significant and tangible 
user commitment in developing the capability may be the best 
measure NASA has for assessing user interest in a lasting pro- 
gram. 


GO/NO-GO DECISIONS 


GO/NO-GO decision points have been included in some pro- 
jects at the end of the "processing" or "demonstration" phase 
and before the start of the "technology transfer" phase. Both 
NASA and the user in these projects have not always taken the 
evaluation and subsequent decision seriously. While optimism 
that a positive "GO" decision will be made is justified, pro- 
ject plans often treat a "GO" decision as a foregone conclu- 
sion. In some cases a significant amount of the evaluation 
work in support of a GO/NO-GO decision has been eliminated and 
the user has less information on which to base his decision. 


If a GO/NO-GO decision is significant enough to be 
required in a project plan, it should be given serious consid- 
eration by both parties. In addition, NASA Headquarters 



should be thoroughly briefed in the justification supporting a 
user's decision. Also, at the time a "GO" decision is made, 
there should be a reassessment and reaffirmation of the user's 
commitments to the project. 

USE OF CONTRACTORS 

Contractors from both the private and academic sectors 
•were utilized in APT and ASVT projects. Private firms parti- 
cipated in these projects both as part of general on-site, 
support service contracts at the NASA field centers and under 
specific contracts to support individual projects. Individual 
contracts were written as both sole source and competitive 
bid. With a few exceptions, user satisfaction with contractor 
support was high for most projects, regardless of the type of 
contract or contractor used. 

One significant trend was evident in the use of contrac- 
tors. Several users had a previously established working 
relationship with a contractor (in many cases a university) 
and an effort was made by the user and NASA to utilize this 
contractor for support in the APT/ASVT project. In these 
cases, users were very satisfied with their contractors and 
thought previous relationships contributed to their ability to 
communicate and work eff iciently . together. From the user's 
perspective, it was worthwhile to utilize these user/contrac- 
tor relationships when they existed. 

NASA COMMITMENTS 


With regard to sustained support for technology transfer 
efforts like the APT and ASVT programs, project managers have 
encountered severe travel limitations, delays in signing 
project agreements, project fund cutbacks at critical times in 



the transfer process, and changes in project personnel or 
reduction in contractor support in favor of higher priority 
programs. These are severe setbacks in projects where user 
confidence and belief in the technology are critical and hinge 
to a great degree on NASA's commitment to support technology 
transfer. Inconsistent support may be permissible in R&D 
activities, but only wholehearted support can consistently 
achieve success in technology transfer projects. 

PROJECT WORKSHOPS 

End-of-pro ject workshops or seminars have been planned 
for several of the projects to serve as technology dissemina- 
tion or transfer mechanisms to other potential users. Those 
workshops which have been conducted to date were open to mem- 
bers of the user community interested in learning about appli- 
cations of the technology, project results, implementation of 
the technology, and opportunities available to them in adopt- 
ing the technology. User participants in the projects in 
which workshops have been or will be conducted are very will- 
ing to share their experiences with other potential users in 
hopes of furthering technology utilization. These workshops 
are a valuable mechanism for technology dissemination when 
they are given wholehearted support. NASA should take advan- 
tage of this opportunity by playing an active role in design 
and conduct of these workshops. 

4.3 FACTORS AFFECTING PROJECT OUTCOME 

In the course of examining the fifteen APT and ASVT pro- 
jects, a number of specific factors have surfaced which have 
had significant impact on the outcome of one or more projects. 


It is unlikely that any of these factors, considered indivi- 
dually, could account for project success or failure. How- 
ever, each factor was observed to have a played a significant 
positive or negative role in some project(s). NASA should 
take advantage of the experience gained from these projects 
and should examine each of these issues in the context of 
future project planning. 

The significant positive and negative factors observed, 
some of which have been previously discussed, are listed 
below. 

POSITIVE FACTORS 

o Project has a strong User Manager 
belief in the program, 
good standing in the user agency. 

o NASA manager has interest in the project and appli- 
cation being studied. 

o User has thorough understanding of Landsat capa- 
bility prior to the project, and/or NASA has realis- 
tically described what the user can expect. 

o Project has adequate and consistent funding and sup- 
port. (e.g. , sufficient travel money for NASA manager 
to participate in important project activities. ) 

o NASA and USER have free exchange of ideas, including 
joint project design. 

o User has examined what is required for operational 

» 

implementation of the technology in his organiza- 
tion. 



o NASA understands the role of Landsat technology in 

the user's activities. 

o User has a mandate (e.g. , legislation, charter) to 

perform on-going, operational activities which can 
utilize Landsat technology. 

o NASA's contractors understand user's needs and the 
two have a good working relationship, (e.g., through 
previous contracts) 

NEGATIVE FACTORS 

o User Agency is not fully committed (e.g., lacks ade- 
quate staff to perform the study or funding to imple- 
ment the technology. ) 

o Project is too time constrained (e.g., project trying 
to meet the needs of operational programs. ) 

o NASA and/or user project managers change during the 
project. 

o Too much political motivation in initiating the pro- 
ject with too little concern for technical merit. 

o Project involves too many agencies and/or disci- 
plines. Project tries to accomplish too much. 

o Users lack faith in Landsat technology (perhaps they 
were oversold in the beginning. ) 

o Conflicts exist at NASA field centers with higher 
priority programs. 



o Internal politics among user agency 
govern project activities (i.e., when 
cies are involved in one project. ) 

o User has immediate, stringent Landsat 
ments. 
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SECTION 5. CONCLUSIONS AND RECOMMENDATIONS 


This section presents conclusions and recominendations 
resulting from this evaluation of APT and ASVT projects. They 
are organized into three categories. General conclusions and 
recommendations about the whole programs and their future 
appear in section 5.1. More specific recommendations about 
the selection and design of future projects are presented in 
sections 5.2 and 5.3. Section 5.2 recommends issues to be 
considered in design of future projects- and section 5.3 
examines considerations for potential user participants and 
applications. 

5.1 GENERAL CONCLUSIONS AND RECOMMENDATIONS 

The following conclusions and recommendations apply to 
the APT and ASVT programs as a whole: 

a. The joint project concept is a viable means to accom- 
plish technology transfer. Both NASA and user parti- 
cipants have expressed their satisfaction with this 
approach. 

b. A wide variety of applications of remotely sensed 
(primarily Landsat) data have been examined in these 
projects. Included are forestry, agriculture, 
hydrology, geology, and urban development. 

c. A broad cross-section of users have been involved in 
these projects. Federal, state, and substate levels 
of government have participated as well as the pri- 
vate sector and the academic community. 



d. The degree of success or level of accomplishments 
varies greatly among projects but significant bene- 
fits have been derived from each one. Benefits range 
from user-supported, operationally-implemented Land- 
sat analysis capabilities to a thorough examination 
and user understanding of an application of remotely 
sensed data. 

e. Approximately one-half of the projects have met or 
appear that they will meet all their stated objec- 
tives. The reasons objectives were not met were 
sometimes within NASA's control and in other 
instances not. (Discussions of these reasons appear 
in sections 3 & 4. ) 

f. The current Landsat data delivery system and uncer- 
tainties about the future of land remote sensing are 
serious impediments to technology transfer efforts. 

g. NASA Headquarters and field centers have provided 
inconsistent support (See Section 4.2) to the APT and 
ASVT programs, which has had a detrimental effect on 
some projects. 

5.2 CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE PROJECT 

PROCEDURES AND DESIGN 

In the planning, design, and conduct of future APT and 
ASVT projects, the following factors are significant and must 
be given serious consideration: 

a. The amount of R&D required for projects must be care- 
fully scrutinized because of its effect on project 
scheduling. When necessary, a pre-APT effort should 


be undertaken to insure that research-proven tech- 
niques are ready for development and testing with a 
user. 

b. Project initiation should be more efficient. The 
signing of project agreements should be expedited to 
get projects underway on schedule. 

c. Project activities should be synchronized to be in 
phase with user activities and requirements, such as 
budget cycles, whenever possible. 

d. The amount of demonstration or test activity (e.g., 
number or size of test sites) must be strictly 
limited to what is required to conduct a valid and 
convincing test. Over-extended project activities 
have resulted in significant schedule slippages. 

e. The scheduling for R&D activities (primarily in APTS) 
should be more flexible than other activities to 
account for uncertainties in the rate of progress. 

f. Serious consideration should be given to GO/NO-GO (or 
redirection) decisions, including formal involvement 
of NASA Headquarters, documentation of rationale 
which forms basis for the decision, and a reaffir- 
mation of user commitments to the project. 

g. Utilizing contractors that have an established 
working relationship with users has been beneficial 
in past projects and should be continued when feasi- 
ble. 



h. Because these projects involve technology transfer 
to outside users, NASA should make efforts to main- 
tain adequate, consistent support, including suffi- 
cient travel money, funding levels which remain as 
originally specified, and stable management and 
staffing levels. 

i. Greater emphasis is needed on separate technology 
"outreach" efforts such as symposia, workshops, and 
conferences, coupled with transferable documen- 
tation. These activities are critical if transfer 
beyond original users is to be accomplished. 

5.3 CONCLUSIONS AND RECOMMENDATIONS FOR EVALUATION OF 

POTENTIAL USER PARTICIPANTS AND APPLICATIONS 

In evaluating potential user participants and their 
resource management applications to be examined in APT and 
ASVT projects, consideration should be given to the following 
issues: 

a. Users should have the responsibility or mandate 
(e.g. , legislative) to perform an on-going, opera- 
tional program which will use the technology once the 
project ends. 

b. Projects should not be undertaken primarily for poli- 
tical reasons. If a project with a particular user 
is politically important, the application to be util- 
ized should be thoroughly examined and understood, 
and the project design tailored to account for speci- 
fic requirements. 
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c. Users should be required to make significantr 
meaning ful commitments (in funds, manpower, equip- 
ment) throughout the projects, in concert with the 

resources available to them. 

0 

d. Applications should be technically sound and not 
place unreasonable demands (e.g., in resolution, 
data delivery) on the current Landsat system. 

e. The number of user entities or groups participating 
in projects should be manageable. If a larger number 
is necessary, a central point of contact with deci- 
sion making authority must be established and main.- 
tained within the user group. 

f. User applications in which the results are to be part 
of an operational user activity should not be 
included in APT or ASVT projects when time 
constraints of the operational activity govern 
project activities. This has proven to be too 
limiting to project flexibility. 



APPENDIX A. NASA MANAGER QUESTIONNAIRE FOR EVALUATION OF 
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QUESTIONNAIRE FOR EVALUATION 
OF NASA'S ASVT AND APT PROJECTS 


PROJECT TITLE 


APT or ASVT 


PROJECT MANAGER/FIELD CENTER 


INTERVIEWER 

OAO Corporation 


DATE 
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A. Project Evolution 


1. Where did the idea come from? 

- NASA HQ 

- NASA Field Center 

- Contractor 

- User Agency 

2. At what level was the project conceived? (e.g., 
administrative, management, working level) 

3. Had there been previous (related) work with the user 

agency (s)? (i.e., to lead into this project) 

4. Were programmatic considerations important in the 
formulation of this project? (as opposed to just its 
technical merit) 

5. What was your first involvement with the project? 

- Previously related work? 

- Project proposal/development? 

- Management after funding approval? 

B. Project Objectives 

6. What do you believe are the objectives of this pro- 
ject? 

7. To what degree have the objectives (both stated and 
your perception) been accomplished? 

8. Have the objectives changed over the duration of the 
project? (e.g., due to over or underestimating the 
degree of success of the project) 
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9. Were the objectives reasonable? (i.e., too difficult 
to attain or not taxing enough) 

10. Do you think NASA and the user agency shared an 
understanding of and appreciation for the goals of 
the project? Was the M.O.U. specific enough? 

Project Participants - Outside private contractor (not 

support service) 

11. Did the outside contractor perform to your satisfac- 
tion during the project? 

12. Did the outside contractor understand the objectives 
of the project? 

13. Was the outside contractor responsive to the needs of 
the user, and were the two able to communicate effec- 
tively? 

14. Did the outside contractor have enough money to per- 
form its tasks properly? (i.e., did any phase of the 
project suffer due to lack of funds?) 

15. Did any unforeseen obstacles arise to hinder the out- 
side contractor in performing its duties? 

16. Would you recommend using an outside contractor on' 
this type of project in the future? 

17. Would the project have been more successful if the 
work had been performed by the field center (or sup- 
port contractor)? 

18. How was the outside contractor selected? Why not an 
on-site support contractor? 



D. Project Participants - User Agency 

19. What was the level of user agency involvement? Did 
they participate to a greater or lesser extent than 
you expected? Why? 

20. How were the user agency participants assigned to the 
project? Were all the "key figures" included? Did 
they have any previous exposure to Landsat/Remote 
Sensing? 

21. Did the "working level" user agency personnel have a 
genuine interest in the project? 

22. Did the user agency personnel have enough time to 
work on the project? 

23. Did the user agency personnel fulfill their obliga- 
tions as far as assignments, deadlines, etc? 

24. Was the user agency prepared to undertake this 
effort? Were they aware of the magnitude of the pro- 
ject and their commitments? Did NASA seem too 
"pushy" in trying to involve them? 

25. What user agency personnel should be interviewed to 
evaluate this project? 

26. How well was this project geared to the user 
agencies' information needs? (e.g., product type, 
format, content) Was there a documented use for this 
technology? 
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Project Participants - NASA HQ 


27. Did NASA HQ provide the proper backing to carry out 
the project? Did NASA HQ impose any constraints on 
you (as a field center) which hindered the project? 

28. Was NASA HQ's interaction (i.e., higher level) with 
the user agency proper? 

29. Did NASA HQ show interest in the project and help you 
solve problems which arose? 

Results/Achievements 

30. Has the technology transfer been successfully 

accomplished? If not, what is the status? 

31. Is the technology developed/transferred currently 
being used by the user agency? 

32. What was the major impediment to the technology 
transfer process? Were you aware of this impediment 
(or its magnitude) at the outset of the project? 

33. How reproducible are the results of this project 
elsewhere? Can this technology be adapted to other 
sites? If not, what factor (s) complicate the pro- 
cess? 

34. Have the results of this project led to the develop- 
ment of other technology transfer efforts in the 
future? 

35. At what stage is the project documentation? Has the 
final report been completed? 

36. Has the work on this project helped identify new R&D 
or technology development needs? 



37. Do you consider the project a success? 


38. Were the training activities sponsored by NASA for 

the user agency valuable in the technology transfer 
process? Was NASA properly prepared to provide 

training? Was the user agency participation 
adequate? 

39. What is your opinion of the cooperative "demonstra- 
tion" project approach? Was user agency participa- 
tion adequate? 

40. In the latter stages of the project, were NASA's 

technical assistance efforts adequate? (as consul- 
tants in the technology implementation at the user 
agencies) Did NASA remain unbiased as far as the 

level and type of user agency involvement? 

41. Did the data processing stage of the project run 

smoothly? Were there any unforeseen impediments 

stemming from data availability or acquisition, 
technique development software or hardware 

unreliability. 

Summary/General 

42. Has the joint approach using NASA and a user agency, , 
employed in ASVT and APT's, been effective? 

43. According to the definitions of APT and ASVT, has 
this project been correctly called an (APT or ASVT)? 

44. What is the most significant result/achievement of 
this project? 



45. How could NASA improve upon the APT/ASVT approach as 
a technology transfer mechanism? 

46. Has the funding been adequate to meet the objectives 
of the project? 

47. Do you feel the user agency has participated at the 
desired/required level? 

48. Did the results of the test or verification phase of 

2 

the project warrant the continuation into the T 
phase? 

49. Was the schedule/timing of the project a problem in 
attaining the objectives? Did the original schedule 
slip? Did NASA rush the project at expense of the 
technology transfer process? Was there enough flex- 
ibility? 

50. Did the user agency appreciate the commitments 
required (e.g., $, manpower, equipment, training) to 
use this technology operationally? 
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QUESTIONNAIRE FOR EVALUATION 
OF NASA'S ASVT AND APT PROJECTS 


PROJECT TITLE 


APT or ASVT 


USER REPRESENTATIVE/AGENCY 


INTERVIEWER 

OAO Corporation 


DATE 
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1. Where did the idea for this project originate? Were 
you contacted by NASA or did the idea come from with- 
in the agency (company)? 

2. Did you have a working relationship with NASA prior 
to the project? 

3. What was the reason for your agency (company) 
examining remote sensing/Landsat technology as an 
information source? (e.g., cost savings, speed, 
accuracy, data format) 

4. How did you become involved with this project as a 
user agency (company) representative? 

5. When did the management level of your agency (com- 
pany) become involved? Were they skeptical at the 
start of the project? 

6. How were the management level personnel kept aware of 
the project accomplishments? Were they interested? 

7. How does this technology compare to traditional tech- 
niques in; 

accuracy 

timeliness 

cost savings (if any) 
information content 

8. Did NASA push to get the project underway before your 
agency (company) was ready? Did the project get 
started according to schedule? 

9. Did the project last longer than the original 
schedule? Why? Was this a problem? 
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10. During the planning and implementation stage of the 
project (prior to the official start) , were you given 
the opportunity to make inputs to the design of the 
project? 

11. Have the objectives your agency (company) set forth 
in this project been accomplished? If not, why not? 

12. Prior to this project, was your agency (company) 
acquiring and using the kind of information derived 
from this technology? (i.e., was this a source of a 
new kind of information or a new way to acquire 
information already being gathered)? 

13. Did your agency (company) appreciate the magnitude of 
its commitment in the go/no-go decision? Were they 
prepared, at the outset of the project, to make the 
"go" decision at the decision point? 

14. Do you think your agency (company) took a risk by 
participating in this project? Were your commitments 
to this project too great? Would your management 
have made a greater commitment? 

15. Was your agency (company) given adequate training in 
understanding and using the technology? 

16. Was your agency (company) given enough opportunity 
for "hands-on" participation in the project work? 

17. Were there any logistical problems (e.g., travel, 
access to equipment, location of personnel) that 
seriously impacted the project? 
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18. Do you think the go/no-go decision was a foregone 
conclusion? Did the evaluation of the technology 
really warrant a "go" decision? 

19. How successful has the transfer of NASA technology 
been to your agency? 

20. Will the information derived from this technology be 
used in conjunction with information from other 
sources? (e.g., in a geographic information system)? 

21. What commitments has your agency (company) made to 
adopt this technology? (e.g., manpower, facilities, 
equipment) 

22. Does your agency (company) have personnel that are 
able to utilize the technology without NASA's help? 

23. Was your agency (company) able to fulfill all its 
commitments throughout the project? If not, why? 
Did NASA meet its obligations? Were the obligations 
of both parties clear at the outset? 

24. Is your agency (company) currently using this tech- 
nology in an operational mode? 

25. Are there any problems involved in using this tech- 

/ 

nology to provide inputs to your resource management 
practices? Is information derived using this tech- 
nology credible and accepted? 

26. Were you satisfied with the performance of NASA's 
contractors (especially outside, not on-site)? Is 
there any reason you would have preferred to work 
directly with NASA personnel? 
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27. At some point during the project, you (the agency or 
company) were asked to evaluate the results of the 
project and make a decision as to whether or not you 
would adopt the technology. Was the evaluation com- 
pleted? Were the results conclusive? Who designed 
the evaluation? How was this technology compared to 
traditional techniques? 

28. What is the status of the project documentation? 
Does the technology used/developed in this project 
exist in a readily transf errable "package"? 

29. Have any post-project activities been initiated to 
extend the technology transfer to other (potential) 
users? 


(for APT) 

30. In many projects of this kind, at least some research 
and development (R&D) work is required. Were you 
surprised by the level of R&D effort required to make 
the technology transferrable? 


(for ASVT) 

31. Was the technology ready to be transferred at the 
start of the project? 
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